Mutual-learning sequence-level knowledge distillation for automatic speech recognition

Abstract Automatic speech recognition (ASR) is a crucial technology for man-machine interaction. End-to-end models have been studied recently in deep learning ASR. However, these are not suitable the practical application of ASR due to their large model sizes and computation costs. To address this issue, we propose novel mutual-learning sequence-level knowledge distillation framework enjoying distinct student structures Trained mutually simultaneously, each learns only from pre-trained teacher but also its peers, which can improve generalization capability whole network, through making up insufficiency bridging gap between teacher. Extensive experiments on TIMIT LibriSpeech corpuses show that, compared with state-of-the-art methods, proposed method achieves an excellent balance accuracy compression.